feat(DLD-1456): add manager-visible runtime recovery endpoints - #2

Align with e2e.yml and ensure CI tests exactly the committed
dependency tree. The pr-policy job already blocks lockfile changes
in PRs, so frozen-lockfile is safe here.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

WIP: routines management, triggers, and execution flow

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

…d and adapterLabels

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…d-home-followups

Improve worktree merge/import followups

…d-home-followups

fix: restore post-merge route verification

…ge-history-migrations

fix: renumber worktree merge history migrations

ci: run e2e tests on PRs

…resh-automerge-guard

fix(ci): skip auto-merge step when lockfile is unchanged

Merge pr-verify.yml, pr-policy.yml, and pr-e2e.yml into a single
pr.yml with three parallel jobs (policy, verify, e2e). Benefits:

- Single concurrency group cancels all jobs on new push
- Consistent Node 24 across all jobs
- One file to maintain instead of three

The jobs still run independently (no artifact sharing) since pnpm
cache makes install fast and the upload/download overhead for
node_modules would negate the savings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…r-workflows

ci: consolidate PR workflows into a single file

…-bootstrap

feat(evals): bootstrap promptfoo eval framework (Phase 0)

…issues-by-project

feat(ui): add project filter to issues list

…image-attachments

fix: embed uploaded images inline in comments via paperclip button

The plugin SDK issues.update() type does not include a comment field.
Post the PR event comment separately via ctx.issues.createComment().

Co-Authored-By: Paperclip <noreply@paperclip.ing>

feat: upgrade to upstream v2026.325.0 + revert npm scope to @paperclipai

…erclipai#86)

* fix(github-plugin): harden delivery dedup, search pagination, and comment fetching

- Replace unbounded per-delivery state keys with a bounded ring buffer
  (200 entries, 24h TTL) stored in a single state key. The plugin SDK
  has no list/scan or TTL support, so individual keys would accumulate
  forever.
- Add configurable maxResults parameter (1-100) to github_search_issues
  tool, defaulting to 10.
- Add automatic pagination to listComments (up to 1000 comments across
  10 pages) so issues with many comments don't silently lose data.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(authz): allow CEO agent to cancel heartbeat runs

Widen POST /heartbeat-runs/:runId/cancel to accept CEO-role agents
in addition to board users. Adds assertBoardOrCeoAgent helper that
does a lightweight role check, company-scoped access guard, and
proper actor attribution in activity logs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

paperclipai#87)

Commit 01c56cd added `"private": true` to all workspace packages during
the @paperclipai_dld → @paperclipai scope rename. This caused the Release
workflow to fail with "no public packages were found in the workspace."
because release-package-map.mjs only considers non-private packages.

Removes the flag from all 12 packages that had it accidentally added,
restoring the pre-merge publish behavior.

Fixes: DLD-1036, DLD-1035

Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>

…paperclipai#88)

The npm registry occasionally returns a 404 for PUT requests on first
publish to a new scope or due to registry propagation lag, even when the
package is successfully queued/published. With set -euo pipefail this
caused the publish loop to abort early, leaving later packages (including
@paperclipai/cli) unpublished and the git tag uncreated.

After a publish failure, we now verify the version is actually available
on npm (6 attempts, 5s delay). If confirmed, we warn and continue. If
not found after retries, we fail with a clear error.

Fixes Release paperclipai#31 CI failure (DLD-1041).

Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>

…params (paperclipai#89)

Switch Step 5 from `pnpm publish` to `npm publish` to avoid pnpm wrapper
issues with scoped package auth. pnpm 9.15.4 passes `--no-git-checks` to
npm which doesn't recognise it, and may handle auth differently for PUT
requests on scoped packages.

Also use NPM_PUBLISH_VERIFY_ATTEMPTS and NPM_PUBLISH_VERIFY_DELAY_SECONDS
(already set in release.yml for CI) as the window for post-failure
verification, matching the patience used in Step 6.

Co-authored-by: Senior Platform Engineer <noreply@paperclip.ing>

…ai#90)

PR paperclipai#87 removed "private": true from 12 packages, causing the release
workflow to attempt publishing to @paperclipai on npm — a scope we
don't own. PRs paperclipai#88 and paperclipai#89 tried to work around the resulting 404
but the root cause is that we don't publish to npm at all.

Restores the invariant documented in CLAUDE.md: all packages use
upstream's @paperclipai scope and are marked private.

Reverts the package.json changes from paperclipai#87.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…inventory

- Add pi-autoresearch team experiment framework (docs/pi-autoresearch-framework.md)
- Document plugin survival across deploys (built-in, volume-installed, CLI extensions)
- Document superpowers skills export procedure
- Document pi-autoresearch extension with reinstall instructions
- Update secrets inventory: add Viracue Stripe keys (test + live), separate
  working vs undecryptable secrets, remove stale entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…inventory (paperclipai#91)

- Add pi-autoresearch team experiment framework (docs/pi-autoresearch-framework.md)
- Document plugin survival across deploys (built-in, volume-installed, CLI extensions)
- Document superpowers skills export procedure
- Document pi-autoresearch extension with reinstall instructions
- Update secrets inventory: add Viracue Stripe keys (test + live), separate
  working vs undecryptable secrets, remove stale entries

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…pattern

The `logChain = logChain.then(…)` pattern in `runChildProcess()` created an
ever-growing linked list of promise closures, each retaining the log chunk
text. For agents producing large output (Research Agent: 257MB in 30min),
this kept all chunks in memory until process exit, causing the server to OOM
at ~4.6GB heap after processing ~35 runs.

Changes:
- Replace promise chain with a shift-based queue that allows GC of processed
  entries immediately after `onLog()` resolves
- Cap `executeProcess()` stdout/stderr buffers at 4MB (previously unbounded)
- Use `.once("exit")` instead of `.on("exit")` in `registerRuntimeService()`
  to prevent listener closure retention after child exit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…mory-leak

fix(memory): OOM crash from unbounded log promise chain

15-18 agents heartbeating simultaneously spawn CLI processes (claude,
opencode, pi) that collectively exceed the 6GB container memory limit.
Each CLI process uses 230-350MB, and the server itself grows to 2.3GB+
under load.

Add HEARTBEAT_GLOBAL_MAX_CONCURRENT_RUNS (default: 3, configurable via
env var). When the limit is reached, excess runs stay queued and are
picked up on the next heartbeat tick (~30s) or when a slot frees up
via the per-agent queue promotion that already runs after each run
completes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…-concurrency-limit

fix(heartbeat): global concurrent run limit to prevent OOM

…pai#94)

The adapter-utils package was never explicitly built in either Dockerfile,
so changes to its TypeScript source (like the PR paperclipai#92 log chain memory fix)
were not compiled into the dist/ used at runtime. The pre-existing dist/
from pnpm install was used instead, containing the old leaking code.

Add `pnpm --filter @paperclipai/adapter-utils build` to both Dockerfiles
before the server build step.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…anup (paperclipai#95)

- Lower --max-old-space-size from 4608MB to 2048MB so V8 GCs
  aggressively and leaves room for child CLI processes (1-1.5GB)
  within the 6GB container limit.
- Reduce plugin session event subscription timeout from 30min to 5min
  to prevent listener accumulation on the live-events EventEmitter.
- Set maxListeners to 50 (was unlimited) so Node.js warns before
  subscriptions spiral.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…pai#96)

When all packages have \"private\": true (as restored in PR paperclipai#90), the
nightly canary release was failing with exit code 1 instead of skipping
cleanly. This caused recurring DLD CI-failure issues (DLD-1201, DLD-1202)
on every scheduled run.

Change the hard fail to a graceful info+exit 0 so the CI job succeeds
as a no-op when this fork has no publishable packages.

Co-authored-by: Senior Platform Engineer <platform@viraforge.ai>
Co-authored-by: Paperclip <noreply@paperclip.ing>

The 2GB limit from PR paperclipai#95 caused 7 restarts: V8 GC stats show the
server legitimately needs ~2GB heap for context resolution, DB queries,
and adapter orchestration across 17 concurrent agents. 2GB left no
headroom for allocation spikes.

3GB balances server needs (2-2.5GB typical, 3GB peak) with child
process memory (3 concurrent × ~600MB = 1.8GB) within the 6GB
container limit.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Exposes process.memoryUsage() and V8 heap space statistics for live
memory monitoring. Heapdump endpoint writes a V8 heap snapshot to disk
for offline analysis of what objects are consuming the 3GB+ heap.

These are essential for diagnosing the remaining memory leak that
persists after the logChain fix (PR paperclipai#92), concurrency limit (PR paperclipai#93),
and subscription timeout tightening (PR paperclipai#95).

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

)

* fix(oom): add /api/health/memory and /api/health/heapdump debug endpoints

Exposes process.memoryUsage() and V8 heap space statistics for live
memory monitoring. Heapdump endpoint writes a V8 heap snapshot to disk
for offline analysis of what objects are consuming the 3GB+ heap.

These are essential for diagnosing the remaining memory leak that
persists after the logChain fix (PR paperclipai#92), concurrency limit (PR paperclipai#93),
and subscription timeout tightening (PR paperclipai#95).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(heartbeat): cap resultJson before DB storage to prevent OOM

Adapters store full stdout/stderr (up to 4 MB each) in resultJson,
which gets written to the heartbeat_runs table.  These multi-MB strings
land in V8's large_object_space (objects >256 KB) and are the primary
driver of the OOM crash loop — production DB shows rows with 4.7 MB
resultJson while large_object_space consumes 669 MB (71% of heap).

Changes:
- Add capResultJsonForStorage() that truncates any string field >8 KB
  to its last 8 KB tail.  Full output is already persisted in NDJSON
  run-log files and stdoutExcerpt/stderrExcerpt columns.
- Apply capResultJsonForStorage at the DB write point in executeRun.
- Null out adapterResult.resultJson after the DB write to allow V8 GC
  before the long executeRun scope ends.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…a, watchdog (DLD-1009)

B1 — Schema migration (0045):
- Add last_heartbeat_at TIMESTAMPTZ DEFAULT NOW() to heartbeat_runs
- Add index on (status, last_heartbeat_at) for efficient watchdog queries

B2 — Ping endpoint:
- POST /api/agents/:agentId/runs/:runId/ping
- Agent JWT auth; caller must own the run
- Updates last_heartbeat_at and returns { lastHeartbeatAt: ISO-8601 }
- 404 if not found, 403 if not owner, 409 if not running

B4 — Watchdog cron:
- heartbeatService.timeoutStaleRuns() — marks running runs timed_out
  if last_heartbeat_at < NOW() - PAPERCLIP_RUN_TIMEOUT_MINUTES (default 30)
- Skips runs with active in-process handles (runningProcesses / activeRunExecutions)
- Releases locked issue execution and promotes next queued run
- Logs to activity_log with actorType=system, action=heartbeat.timed_out
- Wired into existing heartbeatSchedulerIntervalMs setInterval in index.ts

Co-Authored-By: Paperclip <noreply@paperclip.ing>

…gent PASS (DLD-1281)

Adds @paperclipai/plugin-qa-gate: a Paperclip plugin that listens for
issue.updated events and blocks agents from setting status to done unless
a @qa-agent PASS comment exists in the thread.

Bypass rules:
- Board users (actorType=user) always bypass the gate
- Issues labelled no-code, research, docs, or backlog are exempt

10 unit tests pass (createTestHarness). Plugin added to bundled examples
list in server/src/routes/plugins.ts for one-click board installation.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

The POST /api/plugins/:pluginId/enable route was calling lifecycle.enable()
for all statuses, but lifecycle.enable() only accepts disabled/error/
upgrade_pending. Plugins stuck in 'installed' status needed lifecycle.load()
to transition to ready.

Also widens AvailablePluginExample.tag to include "automation" for the
QA Gate bundled plugin entry, and fixes a doc comment that incorrectly
said disable transitions to 'installed' (it's 'disabled').

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Update POST /plugins/:pluginId/enable and /disable to use
assertBoardOrCeoAgent instead of assertBoard, enabling the CEO
agent to call these endpoints without board access.

assertBoardOrCeoAgent already exists in authz.ts and is used
in agents.ts. Added it to the plugins.ts import.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Expand QA gate bypass signals for operational/non-code tickets to avoid auto-reopening stale cleanup issues into in_review. Add unit regressions and Playwright API E2E verification for both bypass and enforcement paths.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Address QA FAIL feedback by adding mandatory plugin version bump (0.1.1), widening stale-operational bypass matching, and updating the Playwright QA-gate flow suite to deterministic harness-based outcome checks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

… fixes

- Add normalizeAgentUrlKey matching in findMentionedAgents so multi-word
  agent names are reachable via @url-key (25/25 agents vs 7/25 before)
- Bump server version 0.3.2 -> 0.3.3
- Fix pre-existing TS errors in heartbeat.ts:
  - setUTCHours extra arg, errorCode out of scope, null assigneeAgentId
- Add in_review assignee validation

Co-Authored-By: Paperclip <noreply@paperclip.ing>

…oken agents

Co-Authored-By: Paperclip <noreply@paperclip.ing>